A structural, content-similarity measure for detecting spam documents on the web

Purpose The Web provides its users with abundant information. Unfortunately, when a Web search is performed, both users and search engines must deal with an annoying problem: the presence of spam documents that are ranked among legitimate ones. The mixed results downgrade the performance of search engines and frustrate users who are required to filter out useless information. To improve the quality of Web searches, the number of spam documents on the Web must be reduced, if they cannot be eradicated entirely. Design/methodology/approach In this paper, we present a novel approach for identifying spam Web documents, which have mismatched titles and bodies and/or low percentage of hidden content in markup data structure. Findings By considering the content and markup of Web documents, we develop a spam-detection tool that is (i) reliable, since we can accurately detect 84.5% of spam/legitimate Web documents, and (ii) computational inexpensive, since the wordcorrelation factors used for content analysis are precomputed. Research limitations/implications Since the bigram-correlation values employed in our spam-detection approach are computed by using the unigram-correlation factors, it imposes additional computational time during the spam-detection process and could generate higher number of misclassified spam Web documents. Originality/value We have verified that our spam-detection approach outperforms existing anti-spam methods by at least 3% in terms of F -measure.